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ABSTRACT 

This  report  examines  the  Navy  Capability  Management  process  that  has  evolved  from 
the  1995  review  of  RAN  logistics  support  and  in  response  to  the  more-recent 
Government  focus  on  output  based  management  of  ADF  assets. 

This  report  seeks  to  show  that  capability  management  should  be  based  on  quantitative 
measures,  including  the  insertion  of  Measures  of  Performance  and  Measures  of 
Effectiveness  at  appropriate  points  in  the  management  process.  Capability 
management  must  also  refer  to  operational  requirements  and  is  therefore  a 
requirements-driven  process.  However,  the  principal  difficulty  with  this  management 
process  is  the  lack  of  processes  and  supporting  tools.  Modelling  and  simulation  are 
presented  in  this  report  as  a  means  of  addressing  this  difficulty.  The  main  purpose  of 
modelling  and  simulation  is  to  predict  the  behaviour  of  ADF  assets  (RAN  platforms 
and  platform  systems)  in  response  to  their  environments.  Environments  in  this  context 
refer  to  changing  operational  requirements  and  constraints  both  within  and  external  to 
the  model. 

Capability  management  is  a  forward-looking,  dynamic  process  and  therefore  there  is  a 
need  to  develop  matching  dynamic  models  as  capability  management  tools.  There  is 
no  single  model  that  can  be  generally  applied  to  the  many  capability  management 
problems  however;  modelling  may  be  applied  effectively  to  address  specific 
management  and  technical  issues.  Models  representing  the  RAN  platforms  and 
platform  systems,  when  applied  in  a  simulation  process  under  a  range  of  conditions, 
can  provide  useful  capability  management  information.  This  information  may  be  used 
to  develop  appropriate  quantitative  performance  measures. 
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The  Role  of  System  Modelling  and  Simulation 
in  Royal  Australian  Navy  Capability 
Management 


Executive  Summary 


This  Report  was  published  previously  as  a  paper  in  Volume  25  of  the  Journal  of  the 
Australian  Naval  Institute.  It  is  published  as  a  General  Document  to  achieve  wider 
dissemination  of  concepts  for  RAN  capability  management  process;  particularly  in  the 
application  of  modelling  and  simulation. 

This  Report  examines  the  navy  capability  management  process  in  mid  1999  and 
discusses  its  development  from  the  1995  review  of  RAN  logistics  support  and  the 
promulgation  of  the  Chief  of  Navy  Preparedness  Directive  in  1998. 

While  capability  management  is  a  business  process,  this  Report  examines  capability 
management  from  a  systems  perspective,  stemming  from  a  basic  definition  of 
capability  stated  in  terms  of  the  ability  of  a  system  to  perform  specific  actions.  The 
management  of  (military)  systems  is  discussed  briefly  in  terms  of  hierarchies  of 
systems  and  the  use  of  Measures  of  Performance  (MOP)  and  Measures  of  Effectiveness 
(MOE)  to  evaluate  the  systems  in  the  context  of  operational  requirements  imposed  on 
them.  The  Report  discusses  the  use  of  modelling  and  simulation  as  a  way  of 
establishing  MOPs  and  MOEs  for  military  capabilities  and  providing  predictions  of 
system  performance  in  response  to  operational  requirements.  Modelling  and 
simulation  is  seen  as  one  way  of  managing  navy  capability  as  a  forward-looking,  pro¬ 
active  process. 

The  Report  briefly  discusses  how  modelling  and  simulation  may  be  applied  to  the 
capability  management  process  and  notes  that  the  models  must  accurately  reflect 
reality,  that  appropriate  data  must  be  collected  and  that  predictions  must  be 
thoroughly  assessed  for  accuracy  and  relevance.  The  Report  argues  that  modelling  and 
simulation  allows  capability  management  decision  making  to  be  made  on  the  basis  of 
many  scenarios,  rather  than  "simple"  historical  data.  Modelling  and  simulation 
therefore  allow  "discovery"  of  problems  and  sensitivities  to  internal  navy  processes 
and  external  environmental  constraints. 

The  Report  briefly  reviews  modelling  and  simulation  techniques  that  have  the 
potential  to  be  valuable  adjuncts  to  the  RAN  capability  management  process.  For 
example,  modelling  and  simulation  have  the  potential  to  advise  decision  making  in 
terms  of  cost/capability  trade-off.  Modelling  and  simulation  tools  and  techniques 
discussed  in  this  Report  include  reliability  block  diagrams,  fault  trees,  multi-criteria 
decision  making,  artificial  neural  networks  and  knowledge  discovery. 


Modelling  and  simulation  need  to  be  developed  specifically  for  the  RAN  capability 
management  processes  and  the  form  of  the  outputs  specifically  tailored  to  the  navy 
business  process.  Modelling  and  simulation  has  the  potential  to  allow  optimisation  of 
capability  delivery  across  navy's  Force  Element  Groups  and  to  assure  materiel 
readiness  for  specific  operational  requirements. 
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1.  Introduction 

Recent  reviews  of  RAN  logistics  support  [1]  and  RAAF  logistics  support  [2]  have 
highlighted  the  need  to  change  business  processes  to  meet  the  preparedness  (readiness 
and  sustainability)  and  supportability  requirements  of  ADF  assets  in  a  resource- 
constrained  environment  [3].  These  reviews  have  identified  that  operational 
requirements  must  drive  the  capability  management  process.  More  generally,  they 
have  identified  significant  gaps  in  the  process  of  linking  technology,  operational 
requirements  and  business  practices. 

The  management  of  RAN  assets  including  human  resources  is  a  complex  process  that, 
over  time,  is  likely  to  increase  further  in  its  level  of  complexity.  One  of  the  reasons  for 
this  increasing  complexity  is  these  RAN  assets  will  not  be  managed  individually,  but  in 
conjunction  with  other  ADF  assets.  To  achieve  effective  financial  and  materiel 
management,  the  RAN  will  need  to  develop  and  apply  new  processes  and  tools  that 
capture  the  dynamics  of  asset  management.  These  new  processes  and  tools  will  be 
used  to  explore  how  the  technical  systems  of  RAN  assets  and  their  in-service  support 
systems  respond  in  a  changing  environment.  The  environment  in  this  context  refers  to 
changing  operational  requirements  and  financial,  technical  and  human-resource 
constraints.  For  example,  how  to  optimise  the  allocation  of  funds  so  that  the  required 
level  of  capability  can  be  achieved  to  perform  a  particular  operational  requirement. 
Another  example  is  when  to  schedule  maintenance  so  that  the  capability  of  a  platform 
is  maximised  during  a  specified  period  of  time.  The  need  for  these  tools  has  been 
recognised  overseas  [4]  and  within  the  RAN  [5,  6].  Significant  effort  within  the  RAN 
has  been  devoted  to  defining  the  framework  for  a  requirements-based  asset- 
management  process  [5,  6],  but  this  process  will  also  need  quantitative  methods  that 
can  be  applied  to  the  dynamic  logistics  and  in-service  support  environments. 

At  present  there  is  no  accepted  definition  of  capability  management.  The  1997-1998 
Defence  Annual  Report  [7]  deals  with  capability  management  in  a  general  way: 

"The  Government  looks  to  Defence  to  provide  the  widest  range  of  effective  military 
options  in  any  military  situation  that  might  arise.  The  Defence  organisation's  most 
important  priority  therefore  is  to  maintain  military  capability  ready  for  operations, 
including,  if  necessary,  for  combat.  That  priority  is  reflected  in  Defence's  output 
structure,  which  identifies  the  elements  of  the  current  capability  as  the  key  products 
provided  to  the  Government." 

In  this  report,  capability  is  defined  as  the  ability  of  a  system  to  perform  specific 
actions.  'System'  is  used  here  as  a  generic  term  that  describes  any  system,  large  or 
small.  Some  examples  of  systems  are  a  Force  Element  Group  (FEG),  platform,  weapon 
system,  propulsion  system,  engine  and  fuel  pump.  In  the  Australian  Defence  Force 
(ADF)  military  capability  is  defined  in  terms  "of  two  elements  -  force  structure  (the 
number,  type  and  grouping  of  military  units,  personnel,  equipment  and  facilities)  and  the 
preparedness  of  that  structure  for  operations"  [8].  An  example  of  how  military  capability 
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can  be  managed  in  an  efficient  and  effective  way  is  given  in  the  Mine  Warfare  Force 
Element  Group  Business  Model  [9].  This  model  defines  the  Mine  Warfare  FEG 
capability  in  terms  of  materiel  support,  operations  and  operational  support  (Figure  1). 
The  activities  or  actions  that  make  up  this  capability  include  maintenance  engineering, 
training,  configuration  management,  managing  preparedness  requirements,  planning 
missions,  conducting  operations  and  providing  port  services. 

Capability  management  implies  that  the  ability  of  a  system  to  perform  specific  actions 
needs  to  be  measured  and  compared  with  a  desired  measure  or  benchmark.  A 
capability  gap  exists  when  there  is  a  deficiency  between  the  measured  ability  of  the 
system  and  the  desired  ability  of  the  system.  Simply  stated,  capability  management  is 
the  process  to  identify  and  address  the  capability  gap  within  the  temporal 
environment.  It  is  a  forward-looking  process,  particularly  when  considering  a  range  of 
possible  operational  requirements.  The  main  three  causes  of  the  capability  gap  are: 

1.  changes  in  operational  requirements, 

2. changes  in  the  environment,  and 

3. changes  in  the  performance  of  the  RAN  assets. 


Figure  1:  The  Mine  Warfare  Force  Element  Group  Business  Model  defines  capability  in  terms  of 
materiel  support ,  operations  and  operational  support. 


The  critical  question  that  should  be  asked  about  capability  management  is  " manage  to 
do  what?".  Capability  management  needs  to  be  driven  by  operational  requirements, 
that  is,  RAN  assets  respond  to  requirements  to  achieve  specific  outcomes.  The  dynamic 
nature  of  the  capability  management  process  will  be  discussed  in  this  report,  and 
following  from  this,  it  will  be  argued  that  effective  management  of  RAN  capability  can 
only  be  achieved  using  quantitative  methods.  The  application  of  system  modelling  and 
simulation  will  be  shown  to  be  one  method  of  quantitative  management.  Modelling 
and  simulation  will  allow  the  generation  of  "virtual  histories"  which  will  allow 
forward-looking  management  decisions  to  be  made  and  which  will  be  directly  related 
to  the  requirements  for  the  capability.  The  reality  of  capability  management  is  that 
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RAN  assets  contributing  to  the  capability  cannot  be  placed  (and  therefore  tested) 
under  all  possible  operational  conditions. 

There  are  many  modelling  and  simulation  tools  currently  available  that  could 
contribute  to  the  capability  management  process.  Tools  based  on  modelling  and 
simulation  need  to  be  initially  developed  to  address  specific  aspects  of  the  capability 
management  process.  This  report  focuses  on  the  materiel  support  aspects  of  the 
capability  management  process  that  are  the  responsibility  of  the  Class  Logistic  Offices 
in  Support  Command  -  Navy.  For  example,  modelling  and  simulation  tools  could  be 
applied  to  resources  allocation,  maintenance  scheduling,  spares  optimisation  and 
obsolescence  management.  The  results  from  these  tools  will  contribute  to  the 
performance  and  effectiveness  evaluations  within  the  capability  management  process. 

In  the  next  section  capability  management  is  discussed  in  terms  of  systems  concepts. 
This  is  followed  by  a  discussion  of  the  need  for  quantitative  processes  in  capability 
management  and  how  modelling  and  simulation  can  be  applied  to  this  process.  Some 
of  the  modelling  and  simulation  techniques  that  could  be  used  are  presented  in  the 
next  section,  followed  by  a  short  summary  of  this  report. 


2.  Capability  Management 

This  section  begins  with  an  explanation  of  system  concepts,  which  is  then  used  to 
define  Measures  of  Performance  (MOP)  and  Measures  of  Effectiveness  (MOE)  in 
Section  2.2.  These  concepts  are  then  used  in  defining  the  capability  management 
process  in  Section  2.3. 

2.1  Systems  Concepts  in  Capability  Management 

In  this  section,  the  definitions  of  a  'system'  and  a  'component'  are  given.  A  more 
detailed  examination  of  these  concepts  and  their  application  to  capability  management 
is  given  in  [10]. 

The  term  'system'  is  quite  loosely  applied  to  many  animate  and  inanimate  objects, 
groups  of  objects  and  ideas.  The  most  helpful  and  general  view  of  a  system  is  that  its 
collective  function  is  only  made  possible  by  reaction  and  interaction  of  its  overall  sub¬ 
systems  and  that  no  individual  system  can  exist  without  some  level  of  interaction  with 
other  systems.  Collective,  interactive  systems  themselves  are  part  of  some  larger 
system;  thus  to  understand  a  system  requires  appropriate  boundaries  be  established 
which  encompass  all  essential  features  contributing  to  the  function  of  the  system  under 
study.  This  view  is  most  instructive  when  dealing  with  management  and  business 
systems  since  the  function  of  a  business  system  cannot  be  assessed  without  reference  to 
other  systems  from  which  it  draws  and  supplies  information  while  (often)  imposing 
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mechanisms  and  constraints  on  other  systems,  including  those  that  provide  or  use 
information. 

"Every  system  is  part  of  a  larger  system.  It  is  a  system  in  its  own  right,  and  it  is  also  a  sub¬ 
system  of  the  larger  system"  [11].  The  purpose  of  the  sub-system  is  to  support  the  larger 
system  in  performing  a  task.  "There  are  endless  chains  of  systems  within  systems,  in 
hierarchical  relationship  to  each  other"  [11].  The  hierarchies  of  systems  for  three  examples 
are  given  in  Figures  2a-c.  In  each  of  these  examples  only  one  system  at  each  level  is 
given.  A  number  of  systems  at  each  level  can  be  combined  to  form  the  system  at  the 
level  above.  In  this  report  the  systems  at  the  lowest  level  in  the  systems  hierarchy  are 
called  'components'.  The  analyst  who  is  studying  the  'total'  system  determines  the 
levels  that  make  up  the  system  hierarchy.  Therefore,  in  the  examples  in  Figures  2a-c, 
the  components  are  transformer,  anti-ship  missile  defence  system  and  fuel  pump, 
respectively. 


•  Government 

•  Department  of  Defence 

•  Navy 

•  Force  Element  Group 

•  Ship  /  Submarine 

•  Anti-Ship  Missile  Defence  System 

•  Radar 

•  Transmitter 

•  Power  Source 

•  Transformer 


•  Nation 

•  Government 

•  Department  of  Defence 

•  Navy 

•  Force  Element  Group 

•  Ship /Submarine 

•  Anti-Ship  Missile  Defence  System 


(b) 


(a) 


•  Government 

•  Department  of  Defence 

•  Navy 

•  Force  Element  Group 

•  Ship /Submarine 

•  Propulsion  System 

•  Engine 

•  Fuel  Pump 


(c) 


Figure  2:  Three  Systems  Hierarchies  examples,  adapted  from  Figure  1  in  [11], 
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2.2  Measures  of  Effectiveness  (MOE)  and  Measures  of  Performance 
(MOP) 

Measures  of  Effectiveness  (MOE)  and  Measures  of  Performance  (MOP)  are  two  sets  of 
metrics  that  can  be  used  in  quantitative  management  processes.  MOEs  are 
measurements  of  the  ability  of  a  system  to  meet  a  requirement.  Requirements  define 
what  a  system  should  be  capable  of  achieving.  The  effectiveness  of  a  system  in 
supporting  the  larger  system  can  only  be  evaluated  by  determining  how  well  it  has 
helped  the  larger  system  meet  its  requirement  [11].  MOPs  are  measurements  of  the 
performance  of  a  system  that  result  from  the  particular  way  it  is  designed.  MOPs  may 
be  performance  characteristics  derived  from  tests  and/or  trials  of  systems. 
Performance  characteristics  can  be  measured  quantitatively  and  assessed  against  a 
baseline. 

To  avoid  confusion  between  MOEs  and  MOPs,  it  is  useful  to  think  of  them  in  terms  of 
the  systems  hierarchy.  It  is  possible  to  determine  a  system's  MOPs  in  isolation  but  the 
MOEs  of  that  system  cannot  be  determined  unless  the  requirements  of  the  larger 
system  it  supports  have  been  specified.  For  example,  the  MOPs  of  the  fuel  pump  in 
Figure  2c  are  inherent  to  the  fuel  pump  but  the  engine  is  the  source  of  the  MOEs  for  the 
fuel  pump.  The  performance  of  the  fuel  pump  may  satisfy  the  designers  and 
manufacturers  specifications  but  its  effectiveness  is  determined  by  how  well  it 
supports  tire  engine  in  the  performance  of  its  function.  The  capacity  of  the  fuel  pump  to 
supply  fuel  to  the  engine  may  be  too  low  for  the  engine's  needs  for  fuel.  Similarly,  the 
performance  of  the  engine  may  satisfy  the  designers'  and  manufacturers'  specifications 
but  its  effectiveness  is  determined  by  how  well  it  supports  the  propulsion  system  in  the 
performance  of  its  function.  This  principle  continues  up  the  system  hierarchy.  The  ship 
or  submarine  is  a  sub-system  of  a  larger  system,  the  FEG.  The  ship  or  submarine  is  not 
working  alone  but  in  conjunction  with  other  defence  assets  in  the  FEG  to  accomplish 
the  FEG's  mission.  Therefore,  the  MOEs  of  the  ship  or  submarine  are  determined  by 
the  FEG's  objectives.  These  examples  illustrate  the  differences  between  MOEs  and 
MOPs  and  these  differences  are  summarised  in  Figure  3.  (Figure  1  in  [12]). 
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Objectives 


Output 

Completed 

Task 


Figure  3:  The  conceptual  difference  between  MOE  and  MOP  as  illustrated  in  Figure  1  in  [12]. 


2.3  The  Capability  Management  Process 

ADF  assets,  specifically  RAN  assets,  exist  to  perform  specific  functions  in  relation  to 
defence  requirements.  The  capability  management  process  enables  RAN  assets  to 
achieve  specific  levels  of  performance  once  they  have  reached  the  operational  phases  of 
their  life-cycles1. 

The  capability  management  process  is  illustrated  in  Figure  4.  This  process  begins  with 
the  mission  statement  that  is  derived  from  Australia's  Strategic  Policy  [3]  and 
translated  into  a  set  of  military  strategies  and  Military  Response  Options  (MROs) 
which  form  the  basis  of  new  preparedness  directives  [7].  The  operational  requirements 
for  the  RAN  are  then  specified  in  the  Chief  of  Navy  Preparedness  Directive  [13]. 

The  management  process  requires  assessment  processes  to  answer  the  questions  "Flow 
well  has  the  current  capability  satisfied  the  current  operational  requirement  and  how  well  will 
the  current  capability  satisfy  future  operational  requirements?".  The  formal  process  for 
verifying  this  is  through  the  Capability  Assessment  Reporting  (CAR)  process  and  the 
Capability  Management  Boards  (CMB)  [14].  The  CAR  and  the  CMB  processes  highlight 
deficiencies  in  capability  however,  quantitative  approaches  have  yet  to  be  put  in  place 
to  support  this  management  process. 

In  the  capability  management  process,  both  the  MOEs  derived  from  the  operational 
requirements  and  the  MOPs  of  the  current  capability  are  used  to  assess  whether  the 
current  capability  can  adequately  meet  the  operational  requirements.  This  assessment 
process  requires  MOEs  and  MOPs  to  be: 

•timely  (that  is,  they  can  be  evaluated  within  a  specified  time  period), 

1  DI(AF)  ADMIN9/98  addresses  capability  management  of  systems  “...  from  development  through  to 
maturity,  and  replacement  where  appropriate...”. 
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•relevant  (that  is,  related  to  requirements  which  may  change  with  time), 

•inserted  at  the  appropriate  points  in  the  management  hierarchy, 

•meaningful  to  the  management  level, 

•recorded, 

•tested  (to  ensure  that  data  are  not  biased). 

If  the  outcome  of  the  assessment  process  is  that  the  operational  requirements  are  not 
met,  or  in  the  future  will  not  be  adequately  met  by  the  current  capability  then 
appropriate  action  needs  to  be  taken.  For  example: 

1. the  capability  may  need  to  be  changed  during  its  life-cycle, 

2.  a  new  capability  may  need  to  be  introduced, 

3.  the  training  may  need  to  be  enhanced, 

4.  the  Integrated  Logistics  Support  (ILS)  may  need  to  be  modified,  or 

5.  the  operational  requirements  may  need  to  be  redefined. 

Decisions  to  change  the  capability  or  introduce  a  new  capability  involve  risk- 
management  decisions,  such  as,  cost/ capability  trade-offs  and  re-evaluation  of 
operational  requirements.  The  greatest  difficulty  in  the  capability  management  process 
is  assessing  how  the  capability  will  meet  operational  requirements  in  the  future.  The 
capability  management  process  is  a  dynamic  process  involving  risk  management  and 
continuous  improvement  as  illustrated  by  the  feedback  loops  in  Figure  4. 
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Figure  4:  The  capability  management  process. 
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3.  Quantitative  Processes  In  Capability  Management 

The  systems  approach  to  capability  management  has  the  potential  to  provide 
quantitative  measures  that  can  be  applied  to  all  levels  of  the  RAN  capability 
management  process.  Specific  issues  that  need  to  be  addressed  for  quantitative 
capability  management  are: 

1.  insertion  of  measurement  points  in  the  organisational  structure  for  capability 
management; 

2.  integration  of  capability  development  and  capability  management; 

3.  ability  to  measure  performance  in  response  to  a  changing  environment; 

4.  risk  management  and  decision  making  processes;  and, 

5.  continuous  improvement  in  the  capability  management  process. 

These  issues  are  briefly  addressed  in  the  following  sections. 

3.1  Insertion  of  Measurement  Points  in  the  Organisational  Structure 
for  Capability  Management 

A  description  of  how  measurement  processes  may  be  embedded  within  the  current 
RAN  organisational  structure  is  given  in  this  section.  The  measurement  points  and 
areas  of  influence  in  capability  management  for  the  Maritime  Commander, 
Commodore  Logistics  Support  (Navy)  and  the  Class  Logistics  Managers  are  shown  in 
Figure  5. 

The  purpose  of  Figure  5  is  to  show  that  capability  management  will  often  require 
different  information  to  be  available  to  the  various  management  levels  in  the  process. 
For  example,  the  Maritime  Commander  is  primarily  concerned  with  the  performance 
of  the  fleet  and  facilities  whereas  the  performance  of  an  individual  component  or 
system  on  board  a  RAN  platform  is  the  concern  of  the  relevant  Class  Logistic  Manager. 
In  this  example,  a  RAN  platform  (which  is  a  collection  of  interacting  systems)  can  be 
managed  individually,  yet  this  platform  contributes  to  higher-level  requirements  in 
conjunction  with  similar  and/or  different  platforms.  Therefore,  it  is  important  to 
determine  the  relevant  information  for  each  managerial  level  and  to  put  in  place  the 
appropriate  MOPs  and  MOEs. 

Informed  decision  making  is  dependent  on  establishing  measurement  points  within 
the  capability  management  process  that  are  consistent  with  articulated  operational 
requirements  for  the  capability.  The  measurement  points  in  Figure  5  that  are  used  to 
show  the  appropriate  MOPs  and  MOEs  need  to  be  inserted  at  various  positions  within 
the  business  processes.  The  purpose  of  these  measurement  points  is  to  set  targets  or 
benchmarks  to  be  achieved  that  enable  the  responsible  managers  to  understand  how 
well  the  process  is  performing  [2]. 
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Figure  5:  The  Hierarchy  of  Capability  Management 


Deciding  where  to  place  these  measurement  points  is  a  critical  issue  in  the  design  of  the 
capability  management  process.  At  all  levels,  the  measurement  points  must  be  capable 
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of  capturing  information  relevant  to  the  operational  requirement  of  the  capability.  The 
three  principal  classes  of  performance  information  (which  will  be  measured  against 
performance  benchmarks)  are: 

1.  technical, 

2.  financial,  and 

3. human  (for  example,  training  and  in-service  support  competencies). 

Measured  characteristics  at  many  of  the  measurement  points  will  be  related.  For 
example,  financial  performance  may  have  a  strong  impact  on  training  and  hence 
human  performance. 

In  many  management  systems  it  is  often  possible  to  find  that  the  performance 
measures  used  are  those  that  are  easy  to  measure.  However,  they  may  not  convey 
information  as  to  how  the  system  is  functioning.  For  example,  from  a  Defence 
perspective,  there  are  few  performance  measures  that  are  related  to  mission  reliability, 
sustainability  and  cost  effectiveness  [2],  This  reference  also  notes  that: 

"...  there  are  no  agreed  benchmarks  for  performance  measures  in  an  operational 
planning  sense.  The  type  of  inf ormation  available  is  historic ,  giving  no  indications  of 
future  trends  or  outcomes...". 

The  result  is  that  operational  requirements  are  usually  adjusted  to  accommodate 
platform  availability  rather  than  operation  requirements  driving  the  availability2. 

3.2  Integration  of  Capability  Development  and  Capability 
Management 

It  is  important  that  the  capability  management  process  is  not  separated  from  the 
capability  development  process.  It  could  be  argued  that  the  capability  management 
process  commences  before  assets  are  acquired.  As  with  all  logistics  processes,  down¬ 
stream  processes  are  affected  significantly  by  decisions  made  in  the  early  stages  of 
capability  development.  There  is  an  obvious  requirement  that  all  of  the  supportability 
(and  configuration)  information  will  be  transferred  to  the  operational  and  support 
phases  of  the  asset's  life-cycle  and  that  it  will  continuously  be  maintained  throughout 
the  life-cycle  of  the  asset.  Assets  are  often  managed  in  an  information-poor 
environment  because  management  processes  are  not  in  place  to  maintain  such 
information  throughout  their  lives.  Additionally,  the  performance  characteristics  of 
assets  are  often  evaluated  under  "test-bed"  conditions  and  therefore  do  not  reflect  their 
performance  within  the  larger  Defence  environment.  Therefore,  management 
processes,  including  sparing,  maintenance  scheduling  and  maintenance  are  often  based 
on  inappropriate  data  which  are  irregularly  updated  (if  at  all).  Further,  new  roles  and 
tasks  that  are  outside  the  asset's  original  design  may  evolve  for  an  asset  after  its 
operational  life  begins.  Processes  need  to  be  developed  to  clearly  state  the  desired 
capability  of  assets  and  their  support  requirements  continuously  throughout  the  life- 
cycle. 


2  Another  issue  is  whether  “availability”  (various  definitions)  is  an  appropriate  performance  measure. 
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This  has  been  partially  addressed  in  two  areas  relevant  to  RAN  assets: 

l.RAN's  Detailed  Operational  Requirements  document  (DOR)  for  assets  is  to 
become  a  "living  document"  which  will  capture  changing  capability  baselines 
in  response  to  changed  operational  requirements  [13].  This  document  will 
become  the  Capability  System  Statement; 

2.1ntegrated  Logistics  Support  (ILS)  arrangements  will  be  maintained  throughout 
the  life  of  the  asset  [15]  that  includes  formal  processes  for  the  through-life 
maintenance  of  the  ILS  documentation  for  ADF  assets.  Collection  and 
management  of  asset  and  in-service  support  provision  performance  data  will  be 
essential  to  the  maintenance  of  through-life  ILS  data. 

3.3  Ability  to  Measure  Performance  in  Response  to  a  Changing 
Environment 

One  of  the  greatest  problems  inherent  in  the  RAN  capability  management  process  is 
that  the  process  is  largely  undertaken  under  peacetime  conditions  and  information 
derived  from  these  conditions  does  not  readily  extrapolate  to  contingency  conditions. 
Information  and  management  processes  from  peacetime  operations  may  become  less 
relevant  and/or  inaccurate  when  in  an  environment  of  increasing  threat  levels. 
Therefore,  the  capability  management  process  must  include  strategies  and 
contingencies  to  ensure  preparedness  of  assets  under  all  credible  operational 
conditions. 

Preparedness  is  defined  in  terms  of  readiness  and  sustainability  [14].  Readiness  implies 
that  ADF  assets  must  be  at  a  particular  level  of  operational  capability.  This  includes 
equipment  and  its  condition,  personnel  and  their  level  of  training.  Readiness  of  RAN 
assets  is  currently  assessed  internally  and  stated  as  Minimum  Level  of  Capability 
(MLOC),  Operational  Level  of  Capability  (OLOC)  and  Present  Level  of  Capability 
(PLOC).  These  "measures  of  capability"  need  to  be  assessed  quantitatively.  Collected 
data  may  be  representative  of  past  and  current  capability  in  response  to  a  range  of 
operational  conditions  but  this  is  only  a  subset  of  all  possible  operational  conditions. 
Unfortunately,  difficulties  still  exist  in  determining  the  MLOC  to  OLOC  dynamic  [8] 
because  principally  there  is  still  no  way  of  quantifying  the  relationship  of  platform 
systems  to  operational  requirements.  The  development  of  suitable  MOPs  and  MOEs 
will  provide  a  mechanism  to  quantify  the  relationships  between  platform  systems  and 
operational  requirements  and  this  is  the  focus  of  current  research  [10]. 

Sustainability  is  the  capacity  to  support  ADF  forces  in  operations  and  includes  the 
adequacy  of  material  support,  relief  of  personnel  and  serviceability  of  assets  whilst 
ADF  forces  are  deployed.  This  includes  the  ability  of  industry  to  maintain  a  surge 
capacity  for  the  continual  supply  of  materiel  and  equipment.  It  is  particularly  difficult 
to  assess  sustainability  because  it  is  an  attribute  of  the  capability  management  process 
that  is  forward-looking  and  must  consider  more  than  one  operational  scenario. 
Sustainability  modelling  presents  significant  challenges  and  further  work  in  this 


12 


DSTO-GD-0244 


area  is  highly  desirable  if  preparedness  decisions  are  to  be  made  based  on  a  high  level 
of  quantitative  information. 

3.4  Risk  Management  and  Decision  Making  Processes 

It  is  also  instructive  to  view  capability  management  as  a  risk  management  process,  that 
is,  the  risk  of  an  action  or  the  risk  of  inaction  to  address  an  identified  shortfall  in 
capability.  This  is  embodied  in  the  CN  proposal  [13]  for  Navy  Capability  Management 
that  states: 

"...  an  essential  feature  [of  capability  management]  is  managing  the  risk  associated  with 
maintaining  a  credible  capability  for  the  prevailing  and  emerging  strategic  situation  and 
environment  of  high  technological  change  and  limited  resources  . 

From  a  risk  management  perspective,  the  management  processes  must  be  forward- 
looking.  Decision  making  in  risk  management  will  need  to  be  undertaken  using 
quantitative  data.  These  data  will  be  used  to  calculate  the  two  components  of  risk:  the 
likelihood  of  an  event  happening;  and  the  consequences  of  that  event.  For  example,  the 
risks  of  using  a  particular  maintenance  or  sparing  policy  could  be  identified  before  the 
policy  is  implemented.  Decisions  in  risk  management  will  often  involve  cost/ capability 
trade-offs  and  formal  processes  such  as  selection  of  evaluation  criteria,  weighting  of 
evaluation  criteria  and  the  use  of  ratios  as  means  of  evaluation  scoring  [16].  Decisions 
may  also  be  made  on  the  basis  of  data  generated  by  modelling  and  simulation  and 
using  decision-making  tools. 

3.5  Continuous  Improvement  in  the  Capability  Management  Process 

The  use  of  quantitative  measures  means  that  the  performance  of  RAN  assets  can  be 
tracked  in  time.  These  data  can  be  used  to  continuously  improve  the  assets  and  the 
management  process.  Most  importantly,  the  quantitative  measures  (MOPs  and  MOEs) 
used  in  the  capability  management  processes  need  to  be  continuously  tested  to 
determine  their  appropriateness  to  the  current  or  "new"  operational  conditions.  It  is 
not  sufficient  to  put  MOPs  and  MOEs  in  place  and  assume  they  will  never  require 
modification.  As  new  assets  are  introduced  into  the  ADF,  new  methods  of  testing  or 
training  may  be  required.  Also,  the  measurement  points  in  the  capability  management 
process  may  either,  not  be  appropriate,  or  not  be  located  in  the  same  place  in  the 
management  process. 

The  level  and  quality  of  services  such  as  training,  refit  capability,  maintenance  and 
facilities  that  are  supplied  by  in-service  providers  will  also  change  over  time.  The 
capability  management  process  must  be  sufficiently  responsive  such  that  it  both 
captures  the  changing  in-service  support  services  and  that  the  process  itself  is 
responsive  to  any  changes. 
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4.  Application  Of  Modelling  And  Simulation  To  The 
Capability  Management  Process 

What  has  been  discussed  so  far  in  this  report  is  the  dynamic  nature  of  capability 
management.  Capability  management  is  principally  concerned  with  how  RAN  assets 
are  going  to  behave  in  the  future.  The  main  purpose  of  modelling  and  simulation  is  to 
predict  the  behaviour  of  systems  when  constrained  to  a  particular  set  of  operational 
conditions.  The  results  of  modelling  and  simulation  for  the  capability  management 
process  are  appropriate  MOPs  and  MOEs  for  all  possible  requirements.  This  will  lead 
to  informed  decisions  in  risk  management,  including  assessment  of  different 
management  strategies  and  the  ability  to  determine  areas  of  deficient  performance. 

There  are  three  main  constituents  required  for  the  successful  application  of  modelling 
and  simulation  to  the  capability  management  process.  These  constituents  are: 

1.  models  must  accurately  reflect  reality3  (this  includes  both  the  equipment  and 

their  operators); 

2. appropriate  data  must  be  gathered  for  input  to  the  models;  and, 

3. predictions  are  thoroughly  assessed  for  their  accuracy  and  relevance. 

The  models  will  be  described  by  entities  (objects),  the  attributes  or  properties  of  the 
entities,  the  functions  performed  by  the  entities,  the  inputs,  outputs  and  states  and  the 
rules  governing  the  interaction  of  the  entities,  including  the  business  rules.  For 
example,  in  the  case  of  the  Maritime  Defence  Assurance  System  (MDAS)  project  [5,  6]  a 
set  of  business  rules  has  been  developed  to  govern  the  interaction  of  entities. 

When  applied  to  capability  management,  modelling  and  simulation  has  the  potential  to 
provide  mechanisms  which  allow  the  exploration  of  the  outcomes  of  plausible  (and  not 
so  plausible)  scenarios,  requirements,  inputs  and  constraints  applied  to  specified 
systems.  The  virtual  history  so  generated  provides  guidance  to  managers  before 
scenarios  are  encountered.  Therefore,  through  the  application  of  modelling  and 
simulation,  the  capability  manager  has  the  opportunity  to  develop  strategies  and 
contingencies  before  a  plausible  scenario  becomes  reality.  Modelling  and  simulation 
will  allow  management  decisions  to  be  made  on  the  basis  of  many  scenarios.  Historical 
data  can  be  used  to  validate  that  the  modelling  and  simulation  process  has  indeed 
predicted  the  outcome  of  the  actual  set  of  operational  conditions.  This  is  one  validation 
technique  (others  are  listed  in  Section  5.5)  and  can  only  be  used  if  the  historical  data  is 
available.  Mean  time  to  failure,  mean  time  to  repair,  power  output  and  charging  rate 
are  all  examples  of  physical  performance  data  captured  by  management  information 
systems  such  as  Anzac  Ship  Maintenance  Planning  System  (AMPS),  Submarine 
Information  Management  System  (SIMS)  and  Submarine  Information  System  (SIS).  The 
data  from  these  platform  information  systems  may  be  used  in  the  modelling  and 


3  The  results  and  conclusions  of  the  modelling  and  simulation  are  not  significantly  affected  by  the 
simplifications  and  assumptions  made  in  the  models. 
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simulation  process.  However,  this  provides  little  guidance  about  how  assets  may 
behave  under  conditions  significantly  different  to  the  conditions  for  which  they  were 
acquired.  The  role  of  modelling  and  simulation  as  the  forward-looking  capability 
management  process  is  summarised  in  Figure  6. 


VIRTUAL 
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MANY 
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RECORDED 

HISTORY 

SINGLE 
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Figure  6:  Modelling  and  simulation  as  the  forward-looking  capability  management  process. 


The  manager  who  uses  modelling  and  simulation  can  better  assess  the  consequences  of 
decisions  compared  to  the  "intuitive"  manager  who  relies  on  personal  experience  and 
the  experience  of  others.  The  manager  who  uses  modelling  and  simulation  is  able  to 
augment  "intuitive"  processes  by  exploring  a  wider  range  of  scenarios  where  no 
experience  has  been  gained.  For  example,  an  acceleration  of  system  usage  may  result  in 
the  "intuitive"  manager  increasing  the  sparing  rate,  but  modelling  and  simulation  may 
show  that,  for  this  scenario,  sparing  rate  need  not  increase4. 

Modelling  and  simulation  provide  methods  that  allow  "discovery"  of  problems  and 
sensitivities  to  internal  RAN  processes  and  external  environmental  constraints,  and 
allow  the  concurrent  investigation  of  many  scenarios  in  the  capability  management 
process.  This  is  highly  important  when  there  is  human  activity  in  the  system,  since  no 
performance  of  human  activity  is  exactly  the  same5. 

Some  of  the  areas  of  capability  management  in  which  modelling  and  simulation  can 
be  used  are:  critical  systems  analysis,  risk  analysis,  reliability,  availability  and 
maintainability  analysis,  obsolescence  analysis;  and  resource  allocation  analysis. 


4  Hypothetically,  under  increased  usage  equipment  is  not  turned  off-and-on,  thus  thermal  fatigue  is 
decreased  resulting  in  a  decreased  sparing  rate. 

5  The  use  of  “standard  operating  procedures”  etc  attempts  to  overcome  this  difficulty  and  often  failures  in 
human  activity  systems  can  be  attributed  to  “non-standard”  procedures. 
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Critical  systems  analysis  involves  identifying  system  failures  that  could  result  in  non¬ 
completion  or  incomplete  performance  of  an  operation  and  additional  risk  analysis 
identifies  failures  that  could  result  in  loss  of  life,  injury  or  illness  and  significant  loss  of, 
or  damage  to,  the  asset.  One  of  the  purposes  of  reliability6,  availability7  and 
maintainability8  analysis  is  to  predict  the  performance  of  assets.  This  analysis,  when 
combined  with  modelling  and  simulation  allows  exploration  of  a  range  of  different 
scenarios,  such  as  the  impact  of  different  maintenance  schedules  and  levels  of  crew 
experience.  Obsolescence  analysis  is  used  to  identify  components  and  systems  that 
present  a  significant  risk  to  a  system  because  of  discontinuance  of  production,  change 
of  specification  or  change  of  manufacturer.  Resource  allocation  analysis  may  include 
optimisation  of  sparing  levels,  crew  training  and  different  maintenance  schedules  for 
changes  in  usage,  operational  roles  and  operating  environments. 


5.  Numerical  Techniques  For  Capability  Management 

In  this  section  some  of  the  numerical  techniques  that  could  be  applied  to  the  capability 
management  process  are  briefly  discussed.  These  include:  modelling  and  simulation 
techniques;  multi-criteria  decision-making  techniques;  artificial  neural  networks  and 
expert  systems;  and,  knowledge  discovery  techniques.  This  is  followed  by  a  brief 
description  of  verification  and  validation  techniques.  The  use  of  these  numerical 
techniques  will  assist  the  capability  manager  in  the  decision-making  process. 

5.1  Modelling  and  Simulation  Techniques 

There  are  basically  three  mathematical  techniques  for  predicting  the  response  of  an 
asset  when  constrained  to  a  particular  scenario  or  set  of  conditions: 

1.  trend  analysis  using  historical  data, 

2. analytic  models  and, 

3. simulation  models. 

The  main  assumption  of  trend  analysis  using  historical  data  is  that  the  asset  being 
analysed  will  respond  in  the  same  way  as  in  the  past.  Therefore,  if  a  future  event  has 
not  also  occurred  some  time  in  the  past  then  trend  analysis  cannot  predict  how  the 
asset  will  respond  to  this  event.  For  example,  the  effects  of  aging  and  the  consequences 


6  Reliability  [17]  is  “the  inherent  characteristic  of  an  item  related  to  its  ability  to  maintain  functionability 
when  used  as  specified”.  This  is  a  function  of  the  item’s  inherent  factors  (eg.  strength),  environmental 
factors  (eg.  influence  of  temperature)  and  operational  factors  (eg.  technical  education  of  users). 

7  The  term  availability  [18]  is  used  differently  in  different  situations.  The  availability  defined  here  is  the 
operational  availability  and  is  “the  probability  that  a  component  or  system,  when  used  under  stated 
conditions  in  an  actual  operational  environment,  will  operate  satisfactorily  when  called  upon”. 

8  Maintainability  [17]  is  “the  inherent  characteristic  of  an  item  related  to  its  ability  to  be  restored  when  the 
specified  maintenance  task  is  performed”.  This  is  a  function  of  the  personnel  factors  (eg.  influence  of 
skill,  motivation),  conditional  factors  (eg.  influence  of  operating  environment)  and  environmental  factors 
(eg.  temperature,  humidity). 
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of  preventative  maintenance  cannot  be  predicted  using  trend  analysis  on  historical 
data.  Analytical  models  use  sets  of  equations  to  describe  the  behaviour  of  the  asset  and 
then  attempt  to  solve  them.  However,  the  equations  involved  (even  for  a  small  system) 
are  extremely  complex  and  therefore  a  number  of  simplifying  assumptions  must  be 
made  in  order  to  obtain  a  solution.  One  of  the  most  common  simplifications  is  to 
assume  a  constant  failure  rate,  otherwise  it  is  impractical  to  solve  the  equations.  For 
example,  the  consequence  of  using  a  constant  failure  rate  is  the  probability  of  a  system 
failure  will  be  the  same  whether  the  system  is  one  month  old  or  a  hundred  years  old, 
and  whether  it  has  undergone  maintenance  or  not.  Simulation  models  use  statistical 
sampling  to  create  events  in  time  and  simulate  the  behaviour  of  the  asset  as  a  function 
of  time  in  response  to  these  events.  Unlike  both  trend  analysis  and  analytical  models, 
simulation  models  consider  all  possible  behaviour  (or  states)  of  an  asset  and  for  this 
reason  alone  is  the  preferred  mathematical  technique. 

5.1.1  Modelling  and  Simulation  Tools 

In  developing  models,  and  applying  simulation  techniques  to  these  models,  it  is 
important  to  define  what  "a  model"  means.  In  the  context  of  this  report,  [19]  provides 
an  appropriate  definition.  A  model 

"...  is  the  explicit  interpretation  of  one's  understanding  of  a  situation,  or 
merely  of  one's  ideas  about  that  situation.  It  can  be  expressed  in  mathematics, 
symbols  or  words,  but  it  is  essentially  a  description  of  entities,  processes  or 
attributes  and  the  relationships  between  them 

One  important  feature  of  the  modelling  process  is  that  it  often  incorporates  the 
modeller's  view  of  the  system  or  situation,  thus  the  modelling  process  has  a  high  level 
of  human  involvement.  While  different  humans  may  produce  different  models,  the 
expectation  is  that  the  model  is  still  an  effective  representation  of  the  system  or 
situation9  under  study.  This  highlights  the  importance  of  verification10  and  validation11 
in  the  modelling  process. 

5. 1.1.1  Reliability  Block  Diagrams  and  Fault  Trees 

Reliability  block  diagrams  [21]  are  a  graphical  representation  of  the  serial  and  parallel 
functional  relationships  that  exist  between  components  and  systems  that  are  required 
for  the  overall  asset's  performance.  If  each  component  and/or  system  can  be 
characterised  by  a  function  describing  its  failure  with  time  then  the  results  from  the 
analysis  will  be  a  time  dependent  function  that  describes  the  reliability  of  the  overall 


9  The  modeller  has  the  discretion  to  draw  boundaries  defining  the  system  or  situation,  thus  influencing  the 
modelling  process.  The  modelling  process  should  therefore  be  approached  from  a  systems  engineering 
perspective  to  ensure  that  the  model  meets  the  customer’s  requirements. 

10  Verification  is  “determining  whether  a  simulation  model  performs  as  intended”.  [20] 

11  Validation  is  “determining  whether  a  simulation  model  is  an  accurate  representation  of  the  real  world 
system  under  study”.  [20] 
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system.  It  can  also  show  what  systems  and  components  are  important  with  respect  to 
failure  and  provide  insight  into  system  behaviour. 

Reliability  block  diagrams  [21]  can  be  directly  converted  into  fault  tree  diagrams, 
which  show  the  logical  connections  between  failure  events  in  relation  to  the  defined 
'top-level'  system  failure.  Fault  tree  analysis  can  be  used  to  quantify  the  'top-level' 
system  failure  probability. 

5. 1.1. 2  Influence  Diagrams  and  Decision  T rees 

Influence  diagrams  and  decision  trees  [22]  are  two  complementary  approaches  that  can 
be  used  for  structuring  decision  problems  and,  depending  on  the  nature  of  the 
problem,  one  approach  may  be  preferred.  The  decision  analysis  tools  have  different 
advantages  for  modelling  complex  decisions.  Influence  diagrams  and  decision  trees  are 
isomorphic,  that  is,  any  properly  constructed  influence  diagram  can  be  converted  into  a 
decision  tree,  and  visa  versa.  An  influence  diagram  is  a  simple  graphical 
representation  of  a  decision  problem.  The  elements  of  the  decision  problem,  that  is,  the 
decisions  to  be  made,  uncertain  events  and  the  value  of  outcomes,  are  all  represented 
by  different  shapes  in  the  diagrams.  These  shapes  are  then  linked  with  arrows  in 
specific  ways  to  show  relationships  between  the  elements.  Influence  diagrams  are  very 
good  for  showing  a  decision's  structure,  but  they  hide  many  of  the  details.  A  decision 
tree  can  be  used  to  show  the  details  that  are  hidden  or  embedded  in  an  influence 
diagram.  A  decision  tree  represents  all  possible  paths  that  a  decision-maker  might 
follow  through  time,  including  all  possible  decisions  and  outcomes  of  chance  events. 


5.2  Multi-Criteria  Decision-Making  Techniques 

There  are  many  multi-criteria  decision-making  techniques  that  could  be  applied  to  the 
capability  management  process.  These  techniques  can  be  used  in  two  different  ways:  to 
capture  the  decision-maker's  preference;  or,  they  can  be  used  in  further  analysis  of 
modelling  and  simulation  outputs.  Two  of  the  more  popular  techniques,  the  Analytic 
Hierarchy  Process  (AHP)  and  Multi-Attributed  Utility  Theory  (MAUT),  are  briefly 
described  in  this  section. 

5.2.1  Analytic  Hierarchy  Process  (AHP) 

AHP  [23]  is  a  multi-criteria  decision-making  technique  that  allows  consideration  of 
both  objective  and  subjective  factors  in  selecting  the  best  alternative.  AHP  is  one  of  the 
most  popular  multi-criteria  decision-making  methodologies  available  today  and  has 
been  used  in  a  wide  variety  of  applications  including  resource  allocation,  predicting 
likely  outcomes,  cost  benefit  analysis  and  supplier  evaluation.  AHP  decomposes  the 
decision  problem  into  a  hierarchy.  The  hierarchy  consists  of  the  general  goals  and 
objectives  at  the  highest  level,  the  next  level  or  levels  down  contain  the  more  specific 
attributes  and  the  lowest  level  consists  of  the  alternatives.  Pairwise  comparisons  are 
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then  made  between  the  elements  of  each  level  in  relation  to  their  parent  level. 
This  produces  the  local  priorities  of  each  element  in  the  hierarchy.  The  local  priorities 
in  the  various  levels  of  the  hierarchy  are  then  used  to  construct  a  composite  (global)  set 
of  priorities  for  the  alternatives  (that  is,  the  elements  at  the  lowest  level  of  the 
hierarchy).  AHP  has  also  been  extended  for  decision  problems  that  cannot  be 
decomposed  into  a  hierarchy  or  where  only  a  partial  hierarchy  exists.  In  these 
problems  there  may  be  dependencies  and  feedback  and  therefore  it  is  more 
appropriate  to  use  a  network  in  replace  of  a  hierarchy.  This  process  is  called  the 
Analytic  Network  Process  (ANP)  and  has  not  been  applied  as  extensively  as  AHP  since 
it  is  a  relatively  new  concept  [23]. 

5.2.2  Multi- Attributed  Utility  Theory  (MAUT) 

MAUT  is  a  technique  that  uses  the  decision-maker's  preferences,  involving 
uncertainty,  risk  and  other  factors,  in  selecting  alternatives.  In  MAUT,  the  decision- 
maker's  preferences  are  captured  in  the  form  of  a  non-linear  utility  function  for  each 
individual  attribute  or  quantitative  performance  measure.  These  single  attribute  utility 
functions  are  then  combined  into  a  multi-attribute  function,  which  is  a  single  index  of 
the  overall  desirability  of  an  alternative.  Probability  distributions  are  used  to  quantify 
uncertainty  in  the  multi-attribute  function.  MAUT  provides  a  method  of  combining 
MOPs  and  other  quantifiable  factors  into  MOEs. 


5.3  Artificial  Neural  Networks  and  Expert  Systems 

Artificial  Neural  Networks  (ANNs)  and  expert  systems  are  often  thought  to  be 
competing  forms  of  artificial  intelligence.  However,  it  is  more  appropriate  to  consider 
them  to  be  quite  different  approaches,  each  with  distinct  strengths  and  weaknesses. 
Expert  systems  depend  on  rules  (IF-THEN)  and  are  good  at  sequential  logic.  Artificial 
Neural  Networks,  on  the  other  hand,  depend  on  examples,  and  are  good  at  pattern 
recognition.  ANNs  are  trained  on  sets  of  data  that  have  known  outputs  and  are  then 
used  to  predict  the  output  of  a  set  of  data  that  was  not  used  during  training.  For 
example,  the  capability  manager  may  be  concerned  with  some  of  the  parameters  that 
might  influence  the  reliability  and  operational  life  of  an  engine,  such  as,  oil  pressure, 
average  temperature,  minimum  and  maximum  temperatures,  acceleration, 
deceleration  and  average  speed.  Given  sufficient  data  sets  on  which  to  "train"  an  ANN, 
it  may  be  possible  to  create  a  reliable  predictor  of  critical  values  of  these  parameters. 

5.4  Knowledge  Discovery 

Although  knowledge  discovery  [24]  is  not  a  modelling  and  simulation  technique,  it 
has  the  potential  to  complement  modelling  and  simulation.  Knowledge  discovery  may 
also  provide  information  that  will  be  used  in  the  validation  and  verification  process  for 
models  and  may  additionally  provide  guidance  on  where  potential  problems  he  in  the 
systems  under  study. 
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Knowledge  discovery  is  an  automated  technique  used  for  die  "discovery"  of  patterns 
or  relationships,  within  data  sets,  that  may  not  be  readily  seen  by  standard  observation 
and  analysis.  Knowledge  discovery  does  not  require  a  priori  knowledge  or  the 
development  of  hypotheses  concerning  the  structure  of  data  relationships.  The  method 
uses  various  algorithms  to  cluster  the  data  or  determine  relationships  and  patterns. 
This  information  may  then  be  used  for  optimising  various  aspects  of  an  asset  or 
alternatively  to  advise  modelling  strategies. 

Knowledge  discovery  is  most  effective  when  applied  to  large  data  sets,  such  as  data 
stored  within  a  data  warehouse.  Thus,  with  more  data,  it  is  possible  to  form  stronger 
relationships.  In  this  area,  the  RAN  has  in  operation  "on-line"  data  collection  and 
management  systems  associated  with  its  assets  (such  as,  AMPS  and  SIMS/SIS)  and 
additional  information  relating  to  maintenance  of  the  assets,  training,  crew 
competencies,  etc  are  maintained  within  these  data  systems.  Provided  this  data  is 
collected  accurately  and  is  well  maintained  it  should  be  possible  to  perform  analysis  on 
data  sets  using  knowledge  discovery  techniques. 

5.5  Verification  and  Validation 

The  final  phase  that  a  model,  using  any  of  the  above  numerical  techniques,  must 
undergo  before  being  accepted  for  general  use  is  that  of  verification  and  validation.  An 
explanation  of  both  these  terms  is  given  below. 

5.5.1  Verification 

Verification  is  determining  whether  a  model  performs  as  intended  [20].  The  following 
techniques  may  be  used  for  verification: 

•  thoroughly  debug  all  subroutines,  systematically  connect  subroutines  into  the  main 
code  and  at  all  stages  test  the  outputs  of  the  main  model; 

•  trace  the  flow  of  logic  through  each  sub-module  and  the  main  model,  and  test  each 
state  that  the  model  can  enter; 

•  start  with  a  simple  model  and  gradually  make  it  as  complex  as  required  rather  than 
starting  with  a  complex  model; 

•  perform  "structured  walk  throughs"  of  the  code  and  have  several  people  read  and 
evaluate  the  correctness  of  each  sub-module  and  the  main  model; 

•  test  the  model's  ability  to  deal  with  "extreme"  conditions; 

•  test  the  model  using  simplified  assumptions  for  which  the  true  characteristics  of  the 
model  are  known;  and, 

•  use  graphical  outputs  if  possible  to  see  the  progression  of  simulation  models. 

5.5.2  Validation 

Validation  is  determining  whether  a  model  is  an  accurate  representation  of  the  real 
world  system  under  study  [20].  The  following  techniques  maybe  used  for  validation: 
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•  the  model  should  seem  reasonable  to  those  people  with  knowledge  about  the 
system  being  modelled; 

•  the  system  being  modelled  needs  to  be  observed  and  analysed  so  that  the  variables 
that  are  used  are  representative  of  what  is  actually  modelled; 

•  test  quantitatively  the  assumptions  used  in  development  of  the  model,  such  as, 
using  "sensitivity  analysis"; 

•  test  the  model's  outputs  with  what  might  be  reasonably  expected  from  the  actual 
system  under  study; 

•  if  possible,  test  the  results  against  other  similar  models; 

•  test  against  historical  data;  and, 

•  use  statistical  tests  to  compare  the  model's  output  with  that  of  the  system  under 
study. 


6.  Summary 


This  report  has  provided  an  overview  of  the  role  of  modelling  and  simulation  in  RAN 
capability  management.  Capability  management  is  a  forward-looking  dynamic  process 
and  as  this  report  has  shown,  needs  to  be  based  on  suitable  quantitative  measures. 
These  measures  include  appropriate  MOEs  and  MOPs  that  inform  the  capability 
manager  on  whether  a  particular  system  or  collection  of  systems  will  be  able  to  satisfy 
current  and  future  operational  requirements.  This  assessment  process  along  with  other 
assessment  processes,  such  as,  cost/capability  trade-offs,  is  part  of  the  capability 
management  process. 

A  number  of  specific  issues  for  quantitative  capability  management  have  been 
addressed  in  this  report.  These  included:  the  insertion  of  measurement  points  in  the 
organisational  structure  for  capability  management;  the  integration  of  capability 
development  and  capability  management;  the  ability  to  measure  performance  in 
response  to  a  changing  environment;  risk  management  and  decision  making  processes; 
and,  continuous  improvement  in  the  capability  management  process. 

Capability  management  is  principally  concerned  with  how  RAN  assets  are  going  to 
behave  in  the  future.  The  application  of  modelling  and  simulation  to  capability 
management  will  provide  a  means  of  predicting  the  future  behaviour  of  RAN  assets 
and  systems.  This  will  enable  capability  managers  to  make  more  informed  decisions  by 
allowing  them  to  evaluate  several  possible  alternatives  before  choosing  their  course  of 
action.  Some  of  the  materiel  support  areas  of  capability  management  in  which 
modelling  and  simulation  can  be  used  are  critical  systems  analysis,  risk  analysis, 
reliability,  availability  and  maintainability  analysis,  obsolescence  analysis  and  resource 
allocation  analysis. 

Some  of  the  numerical  techniques  that  could  be  used  for  capability  management  were 
also  discussed  in  this  report.  These  techniques  and  tools  included  simulation  models. 
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multi-criteria  decision-making  techniques,  artificial  neural  networks  and  expert 
systems,  and,  knowledge  discovery  techniques.  The  simulation  models  could  be  based 
on  either  reliability  block  diagrams  (fault  trees)  or  decision  trees  (influence  diagrams) 
and  the  multi-criteria  decision-making  techniques  could  be  used  to  capture  the 
decision-maker's  preferences  or  to  further  analyse  the  modelling  and  simulation 
outputs.  A  brief  description  of  verification  and  validation  techniques  was  also 
presented. 

In  conclusion,  capability  management  when  based  on  quantitative  measures  has  the 
potential  to  add  significant  value  to  the  decision-making  process  of  the  capability 
manager. 
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